The Effect of Term Importance Degree on Text Retrieval

Various approaches to index term-weighting have been investigated. In fact, term-weighting is an indispensable process for document ranking in most retrieval systems. As well actual information retrieval systems have to deal with explosive growth of documents of various sizes and terms of various frequencies because an appropriate term-weighting scheme has a crucial impact on the overall performance of systems. This paper attempts to investigate the impact of term-weighting parameters used in the most well-known retrieval models. The study has been particularly focused on normalization of term frequency in weighting schemes. A novel factor which is called "term importance degree" has been identified, which can be applied to term-weighting schemes by using several parameters. The calculated correlations between the parameters of weighting schemes confirmed the impact of this factor to increase the performance of text retrieval systems. Two models of term frequency normalization are inserted in a basic termweighting scheme, which shows the importance of terms. The experiments were carried out on the standard test collections which validated by multiple statistical tests. General Terms Information system, Indexing, Information retrieval